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Abstract: This paper introduces a set of services for the creation of on-line surveys, questionnaires, exams 
and self-assessment tests within a virtual community system used in e-leaming settings. The system, called 
“Online Communities”, is a dynamic web application used as platform for blended learning activities by the 
Faculty of Economics of the University of Trento. The functionalities added to the platform are contextualized 
into virtual communities, where activities take place. This allows us to create, submit and analyze questionnaires 
within and/or outside the boundaries of a community. Furthermore, the system allows to manage libraries of 
queries organized by the teacher/administrator of the community according to topics and levels of difficulty, in 
order to produce tests and generate self-assessment sessions on behalf of students. The addition of a questionnaire/ 
poll/self-assessment tool to a virtual community system can add a lot of interesting features, mainly using the 
inheritance mechanism implemented among different hierarchically-connected communities. This paper will 
present these functionalities and highlight one of these services, precisely the one dedicated to self-assessment. In 
the system, self-assessment tests have indeed a double nature: (1) They are useful to students for checking their 
own level of preparation; and (2) They are useful to lecturers as an indicator of the level of preparation reached by 
the students. In order to encourage the use of the system within the whole educational process, and not to limit the 
real use of the service to the period of the final exams only, the possibility has been foreseen for each participant 
to share his/her own test results, comparing them with the other members of the community, and even offering 
real “challenges”, thus transforming passive and “solitary” learning into something much more stimulating and 
“social”. 
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1. Introduction 

The paper presents a new service that has been introduced in our virtual community system called “Online 
Communities”, a dynamic web application that is implemented in educational settings by the Faculty of 
Economics of the University of Trento. The functionalities added to the platform allow us to create, submit and 
analyze questionnaires within and/or outside the boundaries of the virtual community. The new services are used 
in a traditional way, for assessing students, asking the member of communities about some facts/polls, and doing 
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self assessment sessions for participants, and so on. The peculiarities of the service derive on one side from the 
possibility to integrate this service inside the “virtual community” metaphor, especially the possibility given to 
one community to inherit from parent communities. Another set of interesting application fields regards the 
collaboration processes that could happen inside one community among the participants, i.e., to share test results 
among the different participants of a community. In this way, the participants to a community can compare their 
results with the other members of the community, and even offer real “challenges”, thus transforming passive and 
“solitary” learning into something much more stimulating and “social”. 

Online Communities is capable of modelling the organizational structures of an academic institution, making 
it possible to share services and virtual spaces for both formal communities, such as the courses offered by the 
institution, and for groups of people sharing the same interests. The system is designed, implemented and 
managed by the research group. To each community different types of services are associated: 

(1) publication (syllabus, suggested reading, collection of links, wiki, etc.); 

(2) upload/download (diffusion of learning objects, documentation, etc.); 

(3) synchronous communication (chat, video conference, etc.); 

(4) asynchronous communication (forum, mail, memo, noticeboard. etc.); 

(5) control (statistics on accesses, data warehouse of user behaviour). 

(Colazzo L., Molinari A. & Orlandi E., 2007; Bison I., Colazzo L. & Molinari A., 2003) 

Each community has one or more administrators who decide what services to activate and what user rights to 
assign to the various users. The administrators of a community are not necessarily teachers, and this makes it 
possible to use the communities for experimenting didactic models in alternative to traditional lectures as 
approaches based on cooperative learning or learning by project. 

Online Communities has been in function since February 2005 and is used by all faculties of the university, 
at present it has 10,000 users and more than 370,000 accesses per year. 

Recently, Online Communities has been endowed with a family of functions that make generalized feedback 
mechanisms possible. These functions can help to develop services of different nature: surveys, questionnaires, 
virtual elections, exam tests and self-evaluation tests, etc. All these services are based on the fact that users are 
asked one or more questions and they differ in the elaboration of the received answers. 

This paper will discuss more specifically the service concerning self-evaluation tests, with the peculiarities 
gained by the fact of being managed inside the arena of a virtual community, separated but integrated with other 
communities of the system. The paper is articulated in the following sections: 

(1) In section 2 we will describe the general architecture of the services provided by the virtual community 
system; 

(2) In section 3 we will introduce the “questionnaires” service and its feedback methods, illustrating the 
general characteristics; 

(3) In section 4 we will describe in details the self-evaluation tests, illustrating their functionalities and uses 
from the point of view of the student and the teacher. 

2. Online Communities architecture 

Online Communities is a dynamic web application used as platform for blended learning activities by the 
Faculty of Economics of the University of Trento. In contrast with other FMS, such as Moodle and Blackboard, 
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the system is not based on the metaphor of “course” but that of “virtual community”. Online Communities enables 
to create, in a generalized way, different types of virtual spaces called “communities”, to aggregate them into 
larger organizational structures and to endow them with generalized co-operational services. 

The questionnaire service’s architecture is based on a multi-layer logic. There are different modules 
interconnected among others as floors in a tower: data layer, business logic and presentation layer. 

This traditional software architecture has some particularity due to principal metaphor used to create the 
system: the virtual community. This approach differs from that adopted by other authors, for example. Beamish 
(1995) considers the virtual community as a group of people who communicate using computer mediated 
communication tools. The participants of these communities are physically in different places, however, they can 
exchange information on common interest in a communitarian way. Rheingold (1993) considers VCs as emerging 
social phenomena. More relevant to this research is the approach of Jones (Jones S. G., 1995; 1997) who separates 
conceptually the technological structure of VCs (named virtual settlement) from the community itself. In the 
project presented, the authors have a double role: The authors are both designers and developers of the 
technological infrastructure, but on the other hand they are teachers and administrators. In other words, the 
authors are at the same time managers and users of the platform. Therefore, the authors understand the need to 
separate what technologically the platform can do from how the customers use it. As stated, one of the starting 
points of this approach concerns the process of cooperation and exchange inside a VC. The concept of VC could 
be associated to what in this moment is commonly defined as Web 2.0. The term Web 2.0 (Berners-Lee T., 2002) 
was coined following a reflexion on the evolution of the web and the self-selection of the web applications after 
the collapse of dot.com companies (O’Reilly, T., 2005). In any case, Web 2.0 is not the definition of a neo-digital 
structure, but just a label that identifies the evolution of the structure of the pre-existing global net in the “social” 
sense. In such context the user assumes the active role both as an author and a manipulator of the contents of 
others. It is not a new concept of web but a new way of using the web. The authors permeated the paradigm of VC, 
and its implication in e-learning applications, with the collaborative aspects of web 2.0. They therefore integrated 
in the platform not only the participatory level but especially the collaboration among users of the system. 

The decision was to equip the system with some typical services related with questionnaires supplied by 
specialized software platforms, but adapted to the specificity of the virtual community system. In this way, the 
questionnaire services can take advantage of some peculiarities supplied by the community system: 

(1) Users can play different roles inside the community; 

(2) Permissions and roles can be specific for each service; Just as examples of possible granularity, 
questionnaires have different permissions to simple CRUD (create, read, update, delete), as see statistics, use 
libraries, invite users, etc; 

(3) Permissions can be specific for each community: For example, in the system there is the possibility to 
send invitations to external users. It is possible to activate this feature in a community and deactivate it in another 
one; 

(4) Users can be grouped independently from community membership in work areas; 

(5) Communities can inherit users/services/permissions, etc. from multiple parents; 

(6) Services inherit characteristics from other services; 

(7) Roles can have polymorphic behaviors; 

(8) Roles can be delegated to other people. 

Here is a list of main problems that we believe commercial softwares have, and that the authors are able to 
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overcome thanks to the paradigm of virtual communities. 

(1) Users have very simple roles and permissions schemas and cannot play different roles in the same 
community; 

(2) In questionnaire tools, the structure of the “classroom” or “test group” is flat and therefore no 
“propagation” tools can be implemented (e.g., send email to all members of children communities). 

The communities hosted in the system can be aggregated into larger structures with a hierarchic mechanisms 
and infinite nesting levels. The communities can also be aggregated into larger communities disregarding the 
possible position of the hierarchical structure, in a sort of “transversal” link that overcomes the concept of 
“hierarchy” and follows the idea of “mesh”. Thanks to these features, a complex but powerful mechanism of 
propagation of services/roles/permissions/rights/duties can be set among communities of the same branch or of 
different branches. All users are recognized by the system and by the community: People external to the system 
can see public part of the community (services, material, contents, etc.) only if managers allow this (e.g., a 
questionnaire of one community could be opened to external contributions). Last but not least, a VC is the 
container for collaboration processes not limited to educational activities, but for any collaboration activity needed 
in an organization (e.g., research teams, recreation groups, friends, meetings, conferences, secretariats, board of 
directors, colleagues and next social dinner). 

The core of the application is composed by some abstract entities, i.e., VCs as aggregation of people to which 
some communication services are available in order to obtain certain objectives. With this approach, it could be 
possible to represent all the hierarchical relationships between different types of educational communities (such as 
faculties, didactic paths, master degrees, courses, etc.), as any other relationship among communities inside 
organizations. Inside this schema of relationships and hierarchies, the possibility of propagating self assessment 
tools, questionnaires and polls to: (1) all members of a community; (2) all members of a certain role of a 
community; and (3) all members that have in common the subscription of community X, Y and Z. 

These features reveal how flexible and powerful is the metaphor of community and the inheritance 
mechanism, especially in learning and collaborative environments. 

3. The Online Community questionnaires service 

The implemented services that enable us to obtain feedback on the direct interaction with the users, are based 
on an abstract object called “question”. The type of a “question” could be textual or multi-medial and it is used to 
pose to the users a question, an affirmation that has to be validated, etc. For each “question”, it is possible to insert 
textual contents with relative formatting or even images, video and Latex expressions. 

The insertion of contents different from a simple text, made through an advanced editor, refers both to the 
contents of the questions and- as we shall see- to the contents of the answer options. 

Different types of “answers” can be associated to the “questions”. An “answer”, just like a “question”, is a 
multi-medial object. Different forms of “elementary enquiry models” (EEM) derive from the aggregation of these 
two basic objects. Here below we list the more common ones that at present are implemented into the system: 

(1) Multiple choice: the most frequent EEM type, in which the user can chose between various answer 
options, being able to select one or more of them; 

(2) Open answer: to the given question the user answers with an open text. Each question can have one or 
more open answers according to how the question is structured; 
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(3) Numeral: the question has answers with one or more values of a numeral type that can be associated to a 
unity of measure; 

(4) Rating: question that requires an option with an associated scale of evaluation. The minimum and 
maximum values can be personalized, and the headings of the columns can be either numeral or textual; 

(5) Matrix: the question is structured on lines and columns and requires an evaluation scale. Just as for EEM 
rating, the headings of the columns can be either numeral or textual; 

(6) Dropdown: similar to EEM multiple choice, with the difference that the options are contained in a 
drop-down list and it is possible to select only one option. 

An elementary enquiry model can in turn, further to aggregating questions and answers, be aggregated into 
more abstract structures called in the system questionnaires. Within the static questionnaire, the EEM is organized 
in pages so that during on-line compilation the user is given one page at a time, with the possibility of passing to 
another page, if necessary, changing already given answers. Therefore, the pages can be used to separate questions 
with different topics or, simply, to have a better and faster usability of the viewing of the same. In the case of 
random questions and for self-assessment tests, the number of questions per page is, however, pre-defined. 

Elementary enquiry models are also associated with other properties that make various forms of 
categorization possible, for example, it is possible to associate a question with a level of difficulty. 

The value of this property is important above all for the self-assessment tests in which the user can decide the 
level of difficulty in order to evaluate his/her own learning progress and in the random exam tests, since it enables 
the teacher to easily gauge the overall difficulty of the test. 

The elementary enquire models can be used above all to evaluate something or to acquire opinions. The use 
of EEM for one, rather than another aim depends on the value of a specific EEM property. In case they are being 
used to evaluate, each question can have a weight in scores and each answer option a weight in percentage 
indicating, for the EEM in which it is possible to apply it automatically, one or more correct options. The use of 
weight makes it possible to carry out the evaluation automatically. 

Excluding EEM in which an open answer is required and those that do not offer the possibility to be 
evaluated, both in case of single EEM and questionnaires, the evaluation is automatic and is expressed in the 
corresponding statistic reports. The elementary enquiry models can be collected in libraries with a given name. 

This option makes it possible to reuse the EEM for different aims. For example, it is possible to create as 
many libraries as the main topics of a course are. Using these libraries the user can generate in a casual way N 
exam tests that have all the same difficulty level, that contain questions concerning all the topics of the portion 
required by the administrator but which, at the same time, contain a choice of EEM totally or partially different 
one from the other. Once the library is used to carry out an exam test and each elementary enquiry model is 
equipped with the opportune comments, the same library can be used as a basis for self-evaluation tests. 

The questionnaires are the most useful form for aggregating among each other the EEM, yet the service 
integrated into the web platform Online Communities makes it possible to create other instruments for collecting 
data further to the on-line complication and the visualization of the results and statistics, grouped per community 
and per single user. 

The questionnaires can be directed at the users registered into single communities, at users registered into the 
Online Communities and external users. 

Furthermore, it is possible to create personalized lists of users and to invite them, via e-mail, to do the on-line 
compilation of the questionnaires. 
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For each type of questionnaire it is possible to insert a validity date, a maximum time for the compilation (in 
the case of time limited questionnaires) and a scale of evaluation. 

Furthermore, there exists the function of copying entire questionnaires, most useful for using the same 
questionnaires in different communities, not to mention the function of copying and importing some questions 
from one questionnaire to another. 

For all tests contained in the questionnaires it is possible to create the respective translations so that besides 
Italian speakers, other users can also use this service. 

As you can see in the following Figure 1, the user will visualize an available list of surveys (or 
questionnaires) and will be able to compile, in the preferred language, by clicking in the last column on the flag 
symbol of the desired language. The opportunity to use the system through different languages is one of the basic 
characteristics of the Online Communities, the users of which are in some cases foreign students with difficulty in 
comprehending Italian. 



Public Polls 




f 


Compiled Polls 




Poll name 


Start date 


End date 




® Test Poll 


09/03/2009 13:55:00 


11/03/2009 23:55:00 


^ 11 = 



Figure 1 List of public polls 



Furthermore, within the service it is possible to construct templates of questionnaires that can easily be 
shared by various communities and personalized if need to be. The abstract structure of the questionnaires can, 
therefore, be used to implement services with varying characteristics but which, as already mentioned, can 
conceptually be divided into services for evaluation and services for acquiring opinions. Here below is a list of a 
certain number of services active and/or under construction in Online Communities. 

3.1 Services for evaluation 

The evaluation mechanism can be applied in order to judge the progress of single students just like in an 
exam or to enable a single student to evaluate his/her level of preparation. 

The two cases are, therefore, different: In the case of the exam test, for example, the questionnaires have 
properties that make it possible to determine a maximum time for compilation and the control of the compilation 
workstation, enabling access through login and password for external users. 

Finally, in these services it is possible to make paper prints of the summary of the code number of questions 
for each user, so as to have an attest of the exam, in order to avoid possible protests on behalf of the students 
themselves. 

(1) Self-assessment tests: represent training paths in which the user answers one question at a time and 
receives a feedback from the system. This service will be described in more detail in the fourth part of the paper. 

(2) Exam test: contains a certain number of questions previously created “ex novo” or imported from other 
questionnaires and from libraries and are identical for all users. 

(3) Random exam tests: in this case the questionnaire is composed in a dynamic way and is, therefore, 
different for each user who compiles it, both for the proposed questions and the order of the questions themselves. 
The questions of the random questionnaire are, indeed, extracted at random from pre -defined libraries previously 
created, where the number of questions and levels of difficulty are decided by the author of the questionnaire. 

3.2 Services for acquiring opinions 
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In this case the questionnaire is not used for evaluating but for collecting data on the opinions of the 
compilers. In contrast with the questionnaires for evaluating, where the post-compilation treatment of data is 
oriented at measuring the progress of the single user, the most important data are data on aggregate. 

(1) “Poll” offers a single question and enables the user to visualize the result in a graphic form and to obtain 
updating in real time immediately after compilation. 

(2) “Survey”: This is a typical instrument for collecting opinions consisting of several questions and 
submitted to the user in an anonymous form. The collected data is later organized in an exportable flat file so as to 
be ready for a subsequent statistical evaluation. In Online Communities they are used mainly for collecting 
information concerning customer satisfaction but the possibility of being submitted to users external to the 
community makes them suitable for many research activities typical of socio-economic al studies. 

4. Self-evaluation tests 

A powerful tool made available to teachers to improve their didactic activity is one that makes it possible to 
create self-assessment tests. The teacher can use a library of questions employed already for other purposes and 
adapt it or create a new one with specific characteristics: to each question he can also associate, further to a score 
and a level of difficulty, some suggestions for the students to help them to achieve better self-learning. These 
suggestions are visualized after the student has given his/her answer and they can be of two types (see Figure 2): 

(1) General: suitable for EEM of types not automatically evaluable by the system. They always offer the 
same suggestion independently from the given answer. They enable the teacher to indicate, for instance, which 
texts to consult for further documentation on the question or even to insert a brief lesson on the subject. 

(2) Specific: at present available only for EEM with single/multiple choice tests and in dropdown format. 
They make it possible to personalize the suggestions according to the answer selected by the user. Therefore, they 
offer a more detailed formulation of the comments, offering the teacher the possibility to explain why the single 
option is wrong and why, even if it is correct, another would have been better, to give a more in-depth answer or, 
simply to congratulate the student on the correct option chosen. 
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Figure 2 List of compiled tests 



When the student starts the compilation, he/she has the possibility to select the level of difficulty of the 
answers that he will be asked, so as to better gauge the test on the presumed preparation. If he/she should realise 
that he/she has over- or under-esteemed his/her knowledge, he/she could at any time break off the test session and 
start a new one with a different level of difficulty. 

Later, the student can review the memorized tests, cancel or verify the progress of his/her preparation through 
a complete summary table with which to compare each session with the previous one so as to be able to evaluate 
the progress made in function of the number of correct/wrong/partially correct/non evaluable answers and two 
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different scores. 

The first score is calculated simply on the bases of the evaluable questions which have been answered, the 
second (called normalized), normalizing this score on the maximum obtainable marks, and answering correctly to 
all the evaluable questions. 

Each answer given by the students is entered into a database that the teacher can consult. The latter can 
visualise the number of answers selected for each option by the single user, or read the answers to the open 
questions, thus evaluating in real time the level of learning of the whole class (or of the single student) and, at the 
same time, validating the inserted set of questions. On the basis of the feedback thus obtained it will, therefore, be 
possible to gauge the course on the needs of the students and/or introduce variations in “real time”, optimizing the 
productivity of the lessons and, consequently, the level of global preparation of the class (see Figure 3). 

1. The new year's resolutions are bad for your health. According to Mind (a British charity against mental illness) in fact promise to 

themselves to change or improve something from the beginning of the new year could create some disappointments. 

source: II Messaggero 

For these reasons what is the new year's resolution that you will not realize in 2009? 

Number of answers: 70 

New Yeafs Resolutions 

None of these 

I always keep my new year's re... 

I've never done new year's res... 

Find a job 
Finding a new home 
Find a soul mate 

Spend more time with my partne... 

Graduate 
Studying more 
Do more sport 
Losing weight 
Stop smoking 



Figure 3 Example of poll statistics 

Unfortunately, these data have a certain relevance only if a given “critical mass” is reached. It was, therefore, 
necessary to tackle the students’ bad habit of revising their studies and using e-learning instruments only in the 
final phase of the course and immediately before the exam, thus making the obtained data unusable for the 
teacher. 

In order to remedy this problem and inspired by games used by some social networks, the researchers have 
thought of transforming the tests into a game or, better even, into a challenge. Indeed, every student will be able to 
create a test with an established number of questions chosen at random within the libraries used by the teacher for 
the self-evaluation tests and which he/she will not know in advance. Having launched the challenge to a “friend”, 
he/she will then proceed to compile the test and, once the opponent has done the same, he/she will be able to 
verify the obtained score and compare it with the opponent. 

A second type of “game” does not envisage the direct challenge but, rather, a “challenge” among the users of 
the community who are willing to participate. In this case, once the user has accessed the specific area he/she will 
be able to visualize the provisional classification of the scores and repeat the tests as many times as necessary to 
outdo the best among the community members. 




83 



Feedback mechanisms in learning virtual community settings 



5. Conclusion 

The paper has presented the usage of a questionnaire/poll/self assessment tool in a virtual community system, 
looking at the architectural and structural changes introduced by this new approach. This typical LMS-based 
service is well known in e-learning. The application of questionnaire tools to a virtual community system has been 
particularly interesting, due to the extra possibilities added by the metaphor of “community”. In fact, the system 
presented in this paper allows to create hierarchies of communities, transversal links among communities that are 
part of different branches, propagation of services, roles and permissions of users among communities and so on. 
In this settings, the application of a questionnaire tool opens a series of possibilities for the members of the 
community, that are particularly interesting in the authors’ opinion right for the e-learning field. With these aims 
the authors have developed some specific tools that implement forms of interaction among members of the 
community, relying of the questionnaire tools. 
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